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(57 ^ Abstract: The present invention provides a method and apparatus for autofocusing of an optical instrument, especially a micro- 
scope (10 ) xhe autofocus scheme includes a digital gradient filter for generating a focus score from each captured image. Images 
may ^ captured at various positions and the focus scores analysed to determine a focus score maximum. The position correspond- 
° ing to this maximum is then taken as the focus position. The spatial extent of the filter is preferably sellable so that the digital filter 
° settably selects elements of the image depending on their size in the image. Thus, the filtering process is controllable so that it is 
O influenced by elements of the image whose size may be specified by the operator indirectly by way of the spatial extent of the filter. 
^ This allows the exclusion of small or large elements of the image from the autofocusing procedure which the filter should preferably 
^ ignore. The autofocus scheme is particularly useful for autofocusing biological specimens. 
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ROBUST AUTOFOCUS SYSTEM FOR A MICROSCOPE 



The present invention relates to an autofocus system especially to an autofocus system 
suitable for a wide range of different microscope types, for example, but not limited to, a 
5 fluorescent microscope or a phase contrast microscope. 



Technical Background 

AJong with the introduction of high throughput screenings, quantitative 
microscopy is gaining importance in pharmaceutical research. Fully automatic acquisition 

10 of microscope images is an unattended operation coupled to an automatic image analysis 
system allows for the investigation of morphological changes. Time lapse experiments 
reveal the effect of drug compounds on the dynamics of living cells. Histochemical 
assessment of fixed tissue sections is used to quantify pathological modification. 

A critical step in automatic screening is focusing. Fast and reliable autofocus 

15 methods for the acquisition of microscope images are indispensable for routine use on a 
large scale. Autofocus is also a requirement for any fully automated microscope-based 
image processing system that must scan areas larger than a single field. This requirement 
for autofocus may be generated by several factors including mechanical instability of the 
microscope and irregularity of the sample and/or its container, e.g. a glass slide. For 

20 example, thermal expansion could account for several microns of instability in 
microscopes with lamps acting as unevenly distributed heat sources. Mechanical 
instability may also arise from backlash between moving components in the microscope 
stage driving motor and gears. Preferably autofocus algorithms should be generally 
applicable on a large variety of microscopic modes and on a large variety of preparation 

25 techniques and specimen types. Although autofocusing is a long-standing topic in 

literature, no such generally applicable solution is available. Methods are often designed 
for one kind of imaging mode. Further the assumptions made for determining the focal 
plane in fluorescence microscopy are often not compatible with the same in phase 
contrast microscopy. There has been a long felt need for a method which is generally 

30 applicable in light microscopy. 

From Fourier optics it has been deduced that well-focused images contain more 
detail than images out of focus. Conventionally a focus score is used to measure the 
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amount of detail. The focus curve can be estimated from sampling the focus score for 
different levels of focus. Some examples of focus curves are shown in Figs, la to c. Best 
focus is found by searching for the optimum in the focus curve. In a conventional 
approach the value of the focus score is estimated for a few focus positions. Evaluation of 
5 the scores indicates where on the focus curve to take the next sample. Repeating the 
process iteratively should ensure convergence to the focal plane. A major drawback is 
that such optimization procedure presupposes 1) a uni-modal focus function, and 2) a 
broad-tailed extremum to obtain a wide focus range. The example focus curves in Figs, 
la to c show that this does not hold true in general. In reality, the focus curve depends on 
10 the microscope set-up, imaging mode and preparation characteristics. When the assumed 
shape of the focus curve does not match the real focus curve, or when local extrema 
emerge, convergence to the focal plane is not guaranteed, see "A comparison of different 
focus functions for use in autofocus algorithms", Cytometry 1985; 6: pp 81-91, Groen et 
al. 

15 Groen et al. suggest eight criteria for comparing the performance of autofocus 

functions. These are: 1) unimodality, or the existence of a single maximum or minimum; 
2) accuracy, or coincidence of the extremum and best focus; 3) reproducibility, or a sharp 
extremum; 4) range, or the vertical distance over which the function will unambiguously 
determine the direction to best focus; 5) general applicability, or the ability to work on 

20 different classes of images; 6) insensitivity to other parameters, or independence from 
influences such as changes in mean intensity; 7) video signal compatibility, or the ability 
to use the same video signal as is utilized for image analysis; and 8) implementation, that 
is, it should be possible to calculate the function rapidly. Groen et al. concluded that three 
autofocus functions, i.e., two gradient functions and the intensity variance, performed the 

25 best. However, some autofocus functions that performed well on one specimen did not 
perform well on others and the authors cautioned against extrapolating the results to 
other imaging modes and specimens. Under insensitivity to other parameters is considered 
robustness against noise and optical artifacts common to microscopic image acquisition. 
Further, it is preferable to avoid that unimodality of the focus curve is an absolute 

30 necessary requirement because unimodality cannot be achieved in regularly practice. As a 
consequence, the range of broadness of the extremum in the focus curve is of less 
relevance. 
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Most autofocus methods fall into two categories: position sensing and image 
content analysis. Position sensing methods, such as interferometry, require independent 
calibration of the best focus location and, more importantly, a single well-defined surface 
from which to reflect light or sound. In light microscopy there are often two reflective 
5 surfaces, the coverslip and slide. In addition, tissue specimens can have significant depth 
and best focus is not necessarily achieved at the surface of the glass. These problems 
make absolute position sensing methods impractical for use in light microscopy. Image 
content analysis functions depend only on characteristics measured directly from the 
image. Best focus is found by comparison of these characteristics in a series of images 

10 acquired at different vertical positions. This method of autofocus requires no independent 
reference and is not affected significantly by any additional reflective surfaces. Its most 
important limitation is speed, which is dependent on the video rate, the vertical 
repositioning time, function calculation time and search range. 

The uncertainty in applying autofocus test results from one microscope method to 

1 5 another led to the present invention. The development of the present invention included 
exploring autofocus performance in microscopy of fluorescent stained biological 
specimens. The fluorescent signal can be used directly for autofocus. However, problems 
summarized by others, such as Chen (Chen L B: Fluorescent labeling of mitochondria, in 
Fluorescence Microscopy of Living Cells in Culture, Part A, Wang Y L and Taylor D L, 

20 eds. Academic Press, San Diego, 103-123, 1989), including photobleaching and the 

formation of free radicals, singlet oxygen, and heat, can create conditions under which 
minimizing fluorescent excitation becomes critical. The most critical conditions probably 
occur in analyzing live cells. If the signal is weak and antiphotobleaching agents cannot be 
used because of toxicity, the signal could easily be completely lost in the 5-10 video 

25 frames of exposure required for autofocus. In addition, the fluorescence by-products 
themselves are toxic, and excessive exposure could alter the results or damage living 
cells. Therefore, it is desirable to find a non-destructive imaging technique for autofocus. 
With brightfield microscopy, fluorescent stained cells appear unstained, showing very 
little contrast. Phase-contrast microscopy, on the other hand, gives high contrast images 

30 of unstained cells and is more useful for autofocus. It would be preferable if a single 

autofocus algorithm would provide good autofocus performance for both phase contrast 
and fluorescence microscopy. 
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An object of the present invention is to provide an autofocus method which is 
generally applicable in different microscopic modes. 

A further object is to provide an autofocus method especially suited for an 
unattended operational environment, such as high throughput screenings. 
5 A further object of the present invention is to provide an autofocus method 

especially suited for capturing and monitoring images which vary with time. 

Still a further object of the present inventions is to provide an autofocus method 
which is robust against confounding factors common in microscopy, such as noise, 
optical artifacts and dust on the preparation surface. 

10 

Summary of the Invention 

The present invention includes a method of autofocus of an optical instrument for 
viewing an object and having an auto-focusing mechanism; comprising the steps of: 

step 1 : acquiring a first digital image of the object through the optical instrument, 
1 5 the first digital image comprising a plurality of pixels having pixel values, 

step 2: applying a digital filter to at least some of the pixel values of the first 
digital image to obtain a focus score for the image, the digital filter settably selects 
elements of the image depending on their size in the image for autofocusing. 

The present invention includes a method of autofocus of an optical instrument for 
20 viewing an object and having an auto-focusing mechanism, comprising the steps of: 

step 1: acquiring a first digital image of the object through the optical instrument, 
the first digital image comprising a plurality of pixels having pixel values; 

step 2: applying a digital gradient filter to at least some of the pixel values of the 
first digital image to obtain a focus score for the first digital image; wherein the digital 
25 gradient filtering step includes a smoothing operation having a settable spatial extent. 

The present invention also includes an optical instrument for viewing an object 
and having an auto-focusing mechanism, the optical instrument being adapted to acquire 
a first digital image of the object through the optical instrument, the first digital image 
comprising a plurality of pixels having pixel values; and the auto-focusing mechanism 
30 having a digital filter to filter at least some of the pixel values of the first digital image and 
to obtain a focus score for the first digital image, wherein the digital filter is adapted to 
settably select elements of the image depending on their size in the image for 
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autofocusing. 

The present invention also includes an optical instrument for viewing an object 
and having an auto-focusing mechanism, the optical instrument being adapted to acquire 
a first digital image of the object through the optical instrument, the first digital image 
5 comprising a plurality of pixels having pixel values; and the auto-focusing mechanism 
having a digital gradient filter to filter at least some of the pixel values of the first digital 
image and to obtain a focus score for the first digital image, wherein the digital gradient 
filter includes a smoothing function having a settable spatial extent. 

The present invention also includes an auto-focusing mechanism for an optical 

10 instrument, the optical instrument being provided for viewing an object and for acquiring 
a digital image of the object, the digital image comprising a plurality of pixels having pixel 
values; the mechanism comprising: a digital gradient filter to filter at least some of the 
pixel values of the digital image to obtain a focus score for the digital image, wherein the 
digital gradient filter includes a smoothing function having a settable spatial extent. 

]5 The present invention also includes an auto-focusing mechanism for an optical 

instrument, the optical instrument being provided for viewing an object and for acquiring 
a digital image of the object, the digital image comprising a plurality of pixels having pixel 
values; the mechanism comprising: a digital filter to filter at least some of the pixel values 
of the digital image to obtain a focus score for the digital image, wherein the digital 

20 gradient filter is adapted to settably select elements of the image depending on their size 
in the image for autofocusing. 

In the above method, apparatus and mechanism the spatial extent of the 
smoothing function may be manually or electronically settable or a combination of both. 
For instance, the spatial extent may be manually settable. The operator may enter a 

25 dimension of an object in the image to be captured which is to be used for autofocusing 
purposes. Alternatively, a default value may be selected by the apparatus and a focusing 
attempt made. If no suitable focus score is achieved, an alternative spatial extent for the 
smoothing function may be automatically selected by the apparatus. Alternatively, the 
operator may input a range, e.g. 1 to 5 microns and the apparatus selects the spatial 

30 extent of the smoothing function based on a value derived from the range, e.g. the mid- 
value or the lowest value derived from the range. This value may be used for a first 
attempt at autofocusing. If this first attempt is not successful, the apparatus may select 
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another value within the range specified by the operator. The larger the spatial extent of 
the smoothing function the less the noise in the image but also the greater is the chance 
that small objects in the image are not "seen" by the filter and therefore are not used for 
autofocusing. Where these small objects are dust particles, the failure of the filter to see 

5 these particles is an advantage. Hence, a larger value of the spatial extent can eliminate 
erroneous results caused by "noise", e.g. dust particles. The spatial extent should not be 
chosen too large otherwise the objects which are to viewed in the image may be smaller 
that the spatial extent of the smoothing function with the result that the details of the 
sought object no longer drive the convergence of the filter on the correct focus position. 

10 The ability to manually or electronically select the spatial extent of the smoothing 

function has the advantage that the optimum smoothing extent can be chosen which 
reduces noise to a minimum while still allowing the autofocusing system to select the 
focus position based on the object to be captured in the image. 

The present invention also includes a method of autofocus of an optical 

15 instrument for viewing an object and having an auto-focusing mechanism, comprising the 
steps of: 

step 1 : acquiring a first digital image of the object through the optical instrument, 
the first digital image comprising a plurality of pixels having pixel values; 

step 2: applying a digital gradient filter to at least some of the pixel values of the 
20 first digital image to obtain a focus score for the first digital image; wherein the digital 
gradient filter includes a smoothing operation limited in spatial extent in that it extends 
over a distance smaller than or equal to the image size and extends at least over three 
pixels either side of a pixel whose value is being filtered. The larger spatial extent of the 
smoothing function of the present invention (7 pixels) compared with the spatial extent of 
25 known autofocus gradient filters provides a more robust autofocusing method which is 
able to resolve focus positions accurately and quickJy with specimens which are difficult 
to focus with conventional filters. 

The present invention also includes an optical instrument for viewing an object 
and having an auto-focusing mechanism, the optical instrument being adapted to acquire 
30 a first digital image of the object through the optical instrument, the first digital image 
comprising a plurality of pixels having pixel values; and the auto-focusing mechanism 
having a digital gradient filter to filter at least some of the pixel values of the first digital 



O075709A1 I > 



"wo 00/75709 PCT/EPOO/04987 

7 

image to obtain a focus score for the first digital image, wherein the digital gradient filter 
includes a smoothing function limited in spatial extent in that it extends over a distance 
smaller than or equal to the image size and extends at least over three pixels either side of 
a pixel whose value is being filtered. 
5 The present invention also includes an auto-focusing mechanism for an optical 

instrument, the optical instrument being provided for viewing an object and for acquiring 
a digital image of the object, the digital image comprising a plurality of pixels having pixel 
values; the mechanism comprising: a digital gradient filter to filter at least some of the - 
pixel values of the digital image to obtain a focus score for the digital image, wherein the 
]0 digital gradient filter includes a mathematical smoothing function limited in spatial extent 
in that it extends over a distance smaller than or equal to the image size and extends at 
least over three pixels either side of a pixel whose value is being filtered. 

The digital gradient filter of the method and system in accordance with the present 
invention may be defined by a mathematical function which includes both a differential 
1 5 operator and the smoothing operator. For example, a digital gradient filter in accordance 
with an embodiment of the present invention may be defined by a function having a 
negative and positive lobe around the spatial origin thereof, the mathematical function 
being limited in spatial extent over a distance smaller than or equal to the image size and 
extends at least over three pixels either side of a pixel whose value is being filtered, and 
20 having only one zero crossing within the spatial extent. The function is preferably the first 
spatial derivative of a Gaussian function but the present invention is not limited thereto. 
The present invention also includes higher order differential filters, e.g. Laplacian 
operators. The gradient filter in accordance with the present invention may be unimodal 
but the present invention is not limited thereto. 
25 Generally, with all embodiments of the present invention, the spatial extent of the 

gradient filter in accordance with the present invention determines how noise is excluded 
from the autofocusing procedure using the captured and filtered image. A real image may 
include large objects of interest embedded in a matrix of small objects. During filtering of 
the image, the small objects will act like noise. If a fine grained autofocusing technique is 
30 used, the autofocus mechanism will always focus on the small objects. Preferably, the 

spatial extent of the gradient filter in accordance with the present invention is settable by 
the operator and/or is electronically settable. By altering the spatial extent of the gradient 
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filter in accordance with the present invention, a range of objects in the image from small 
to large may be selected by the autofocus filter. That is the contribution of small or large 
details in the image to the focus score is more or less depending upon the spatial extent of 
the filter. For instance, the present invention includes changing the value of the spatial 
5 extent of the gradient filter if a suitable focus score is not obtained at a first setting of the 
spatial extent. Further the present invention includes filtering each image with several 
different spatial extents for the filter in parallel. Thus, for each image captured during the 
autofocus scan, several focus scores are obtained simultaneously by parallel processing of 
the image data with several different spatial extents. At the end of the travel of the object 
10 to be viewed a plurality of focus curves are obtained from which the best may be selected. 
The present invention also includes other methods of obtaining a suitable focus score 
when an initial value at initial settings is not obtained. For instance, if a suitable focus 
score is not obtained initially the method and system in accordance with the present 
invention includes one or more refocusing attempts in specimen fields adjacent to the field 
] 5 in which good focusing was not obtained. The present invention also includes calculating 
a plurality of focus scores for each image using a different spatial extent for each 
calculation. To save time this may be done in parallel. The present invention also includes 
selectively setting the range of the values of the spatial extent used for each image. By 
specifying a range, the autofocus system and method according to the present invention 
20 does not try to focus on items which are too large or too small. 

It is not anticipated that specific implementation of the digital gradient filter in 
accordance with the present invention is limiting on the present invention. For example, 
the digital filter may be implemented as a convolutional filter n ter Haar Romey BM, 
"Geometry driven Division in Computer Vision", Boston, Kluwer academic Publishers, 
25 1999, page 439", a recursive filter (van Vliet et. al. "Recursive Gaussian derivative 
filters", Proc. 1CPR '98, EEEE Computer Soc. Press, 1998, pp. 509-514), a 
morphological filter (van den Boomgaard et. al. "Quadratic structuring and functions in 
mathematical morphology", Mathematical morphology and its application to signal 
processing", vol. 3, 1996) or similar. 
30 The auto-focusing system, mechanism and method in accordance with the present 

invention may find advantageous use in a microscope. 

The dependent claims define discrete embodiments of the present invention. The 
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present invention will now be described with reference to the following drawings. 

Brief description of the drawings 

Figs, la to c are examples of measured focus curves for a) a bright-field image of 
5 stained neuronal tissue from mice, b) an image of fluorescent beads, and c) a phase 
contrast image of living PC 12 cells, a rat pheochromcytoma cell line. 

Fig. 2 is a schematic representation of a microscope system to which the present 
invention may be applied. 

Figs. 3a to e are schematic representations of mathematical functions which may 
10 be used to define the gradient filter in accordance with embodiments of the present 
invention. 

Figs. 4a to f show the mean, minimum and maximum focus scores (arbitrary units) 
as function of the z position (optical axis of the microscope) measured in accordance with 
a method of the present invention, a) quantitative neuronal morphology, b) cardiac 

15 myocyte dedifferentiation, c) immunohistochemical label detection, d) C. Elegaiis GFP- 
VM screening, e) and f) immunocytochemical label detection, nuclei and immuno signal, 
respectively. The measured focus curves indicated by "max" and "min" represent the 
focus events resulting in the lowest and highest maximum score which indicates the 
variability and influence of noise on the estimate of the focus score. 

20 Fig. 5 is a representation to show the effect of different spatial extents for the 

gradient filter in accordance with an embodiment of the present invention. 

Description of the illustrative embodiments 

The present invention will be described with reference to certain drawings and 
25 embodiments but the invention is not limited thereto but only by the claims. The present 
invention will also be described with respect to a microscope system but the present 
invention is not limited thereto but only by the claims. For instance, the present invention 
may find advantageous use in any optical instrument in which autofocus is of importance. 
Further, the present invention will be mainly described with reference to a microscope 
30 with a specimen stage moveable stage along the optical axis of the microscope but the 
present invention is not limited thereto but includes optical instruments with which 
focusing is obtained by adjustment of an objective of the optical instrument and not by 



BNSDOCID: <WO 0075709A1 J_> 



WO 00/75709 PCT/EPOO/04987 

10 

movement of the specimen. The present invention also includes combinations of the two, 
e.g. adjustment of an objective within a first range and if a suitable focus score is not 
obtained, movement of the specimen stage along the optical axis followed by a further 
focusing attempt. 

5 Fig. 2 illustrates an optical instrument, a microscope system 1, to which the 

autofocus system according to the present invention may be applied. The hardware 
components of the system 1 include a microscope 1, a motorized stage 2 controlled by a 
pair of XY motors 3 and a Z motor 5, an XYZ stage controller 6, a video camera 7, an 
image processor and host processor 9 with a video frame grabber 8. A separate image 

10 processor including video frame grabber may be provided but the present invention is not 
limited thereto. The XYZ stage controller controls the movements in the X, Y, and Z 
directions independently. The Z direction is along the optical axis (the focusing axis) of 
microscope 10. Typically, the microscope stage 2 will be moved laterally and vertically 
under computer control by stepper motors or DC servomotors. Suitable components are 

1 5 further described in detail below in the description of the examples. The specimen 4 to be 
viewed through the microscope 10 is located on the stage 2. Lamps, e.g. fluorescent 
lamps, and other optical accessories well known to the skilled person will not be 
described. 

Although the present invention will mainly be described with reference to an XYZ 
20 stage 2 (movements in three orthogonal directions) the present invention is not limited 

thereto. To obtain focusing there is only a requirement for relative movement between the 
objective 1 1 of the microscope 10 and the specimen 4. It is not anticipated that the 
method of achieving this is a limitation on the present invention. 

The present invention also includes optical devices in which an objective 1 1 is 
25 adjusted to determine a focus position rather than moving the specimen along the optical 
axis. In such an optical instrument the objective 1 1 may be adjusted by any suitable 
adjusting device which may be controlled by the host processor such as a piezoelectric 
objective positioner. The positioner may be sandwiched between the objective turret and 
the objective 11. Such a positioner may be obtained from Polytech, PI, Costa mesa, 
30 California, USA, e.g. an E-810.10 closed loop controller. With such a system the 
objective 1 1 is adjusted instead of moving the specimen along the optical axis of the 
microscope. Combinations of objective and specimen movement are also included within 
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the scope of the present invention. 

In accordance with the present invention measurement of the focus score can best 
be based on the energy content of a linearly filtered image. Further, an optimal focus 
score is preferably output by a gradient filter. 
5 In accordance with the present invention the gradient filter is limited in its spatial 

extent either side of the pixel of the captured image which is being processed. In digital 
filtering the pixel value to be filtered is combined in some way with pixel values from 
pixels around the pixel being filtered. The smoothing function has a spatial extent, i.e. it is 
a function of distance and determines whether or not the value of a pixel at a certain 

10 distance from the pixel being filtered should be included in the filtering for that pixel and 
if so with what weighting. Experiments have shown that this spatial extent should not be 
less than three pixels either side of the pixel being filtered. A reduced spatial extent may 
results in less accurate autofocusing or even in no ability to find a focus position with 
some types of image. The gradient filter may include one or more operators. For instance, 

15 the gradient filter may include at least one of the following: 

1) a gradient operator to generate the first or higher order spatial differential of the image 
followed by a smoothing operator which determines and limits the spatial extent of the 
digital filtering either side of the pixel being filtered. 

2) a combined gradient and smoothing operator which carries out both gradient and 
20 smoothing operations in one pass. 

3) a smoothing operator applied to the image to limit the spatial extent of the pixels 
involved in the filtering around each pixel being filtered followed by a gradient operator 
to generate a first order or higher order spatial differentiation of the smoothed image. 

In accordance with embodiments of the present invention the digital gradient filter 
25 may include a mathematical function having a negative and positive lobe around the 

origin thereof, the mathematical function being limited in spatial extent in that it extends 
over a distance smaller than or equal to the image size and extends at least over three 
pixels either side of a pixel whose value is being filtered. Preferably, the function has only 
one zero crossing within the spatial extent. 
30 Examples of the types of functions useful in embodiments of the gradient filter in 

accordance with the present invention are shown in Figs. 3 A to E. Fig. 3 A shows a 
function with its X axis as the spatial axis in pixel units and its Y axis being the weighting 
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factor used in the digital filtering. The spatial origin preferably coincides with the pixel to 
be filtered (as shown). The function is the spatial derivative of a normal Gaussian curve. 
It can be seen that the function has two lobes, one positive and one negative either side of 
the spatial zero. These lobes extend over a spatial distance of at least 3 pixels either side 

5 of the spatial origin. In fact, in the example shown, the function has appreciable values up 
to 7 pixels each side of the origin. The effect of the difference in sign of the lobes each 
side of the spatial origin is to determine a gradient of the image when the function is used 
for digital filtering. Within the extent of the function which has appreciable values there is 
only one zero crossing. This is preferably at the spatial origin as shown in Fig. 2 A, i.e. the 

10 zero crossing coincides with the pixel to be filtered. 

The present invention is not limited to a derivative of the Gaussian curve as the 
function defining the gradient filter. As shown in Fig. 3B an odd-order Bessel function 
may also be used (the figure shows a first order Bessel function) or any other function 
which has positive and negative lobes either side of the spatial origin. As can be seen from 

1 5 Fig. 3B, the first order Bessel function has two lobes, one positive and one negative, 

either side of the spatial origin. Due to the fact that a Bessel function has more than one 
zero crossing point, the spatial extent of the Bessel function is preferably limited (e.g. by 
truncation) to the distance either side of the origin which lies within the first non-origin 
zero crossing points, for example in Fig. 3B this would mean truncating the Bessel 

20 function at about the fourth pixel so that values of the function at higher pixel values (or 
lower negative values) are zero or negligible. This is shown by the dotted line. 

An alternative approach would be to damp the Bessel function so that the number 
of zero crossings is reduced. The function of Fig. 3C is a dampened version of the first 
order Bessel function of Fig. 3B using the damping function x 2 + 1 . It can be seen that the 

25 effect of the damping is to reduce the Y axis values effectively to zero beyond the fourth 
pixel. Accordingly, the damping function effectively truncates the Bessel function so that 
there is only one zero crossing within the range ± 3 pixels and values at higher pixel 
values are effectively zero. 

The digital gradient filter according to the present invention is not limited to one 

30 dimensional filtering as would be indicated by the filtering functions of Figs. 3A to C. A 
three dimensional representation of a two-dimensional gradient filter function in 
accordance with an embodiment of the present invention is shown schematically in Fig. 
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3D: Fig. 3D shows a two-dimensional spatial differential of a two dimensional normal 
Gaussian curve. The X and Y-axes lie in the plane of the captured image and have pixel 
numbers as units. The Z-axis represents the weighting applied to the relevant pixel 
number as used in the filtering process. The origin 0,0,0 lies on the pixel to be filtered. 

5 The function has two three-dimensional lobes, one positive and one negative. The origin 
of the two lobes preferably coincides with the pixel to be filtered. Rotation of the 
orientation of the lobes as shown schematically in Fig. 3E is also included within the 
present invention and does not reduce the effectiveness of the filtering. 

In accordance with one preferred embodiment of the image filter of the present 

10 invention a two-dimensional Gaussian derivative as shown in Fig. 3D is used within the 
plane (x, y co-ordinates of an x,y,z co-ordinate system in which the direction z is 
perpendicular to the specimen to be imaged, i.e. the direction z is in the focusing 
direction) of the image to measure the focus score. The a of the Gauss filter determines 
the extent of the filtering and is related to the scale of prominent features 

1 5 A suitable focus function is: 



^ x 9 y 



NM ^ } 

x,y 

where/fx,^) is the image gray value, Gx(x,y, a) and Gy(x,y y o) are the first order Gaussian 
derivatives in the x- and v-direction at scale a, NM is the total number of pixels in the 
20 image, and J5r, fy are the image derivatives at scale o in the x- and v-direction, 

respectively. Further, discussion of Gaussian gradient filters may be found in the article by 
ter Haar Romey BM, "Geometry driven Diffusion in Computer Vision", Boston, Kluwer 
academic Publishers, 1999, page 439; and in "Traitement de l'image sur micro- 
ordinateur", Jean- Jacques Toumazet, Sybex, 1987, pages 156 to 158 which describes the 

25 method of J. F. Canny. 

Often, a trade-off between noise sensitivity and detail sensitivity can be observed 
for a specific microscope set-up. For example, in fluorescence microscopy the signal to 
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noise ratio (SNR) is often low, and relatively smooth images are examined. For phase 
contrast microscopy, SNR is high, and small details (the phase transitions) have to be 
detected. Accuracy of autofocusing depends on the signal to noise ratio as propagated 
through the focus score filter. In accordance with an embodiment of the present 
5 invention, the o of the Gaussian filter may be freely chosen by the operator so that it lies 
within the range of 3 pixels to the size of the image. The value of a is preferably chosen 
such that noise is maximally suppressed, while the response to details of interest in the 
image is preserved. For bar-like structures, the value of c preferably conforms to 

o>-^ (2) 
2V3 

10 where the thickness of the bar is given by d. Assuming that the smallest detail to be 
focused may be considered bar shaped, eg. 2 gives an indication for the value of o. 
The focal plane of the microscope is assumed to be within a pre-defined interval Az 
around the start z-position z. The scanning stage is moved down to the position zmin = z - 
l / 2 Az. Backlash correction is applied by always approaching a focusing position from the 

15 same direction, e.g. sending the stage further down than necessary, and raising it again to 
the given position. As a result, mechanical tolerance in cogwheels is eliminated. 

As / = 0 ms, the stage controller starts raising the stage to traverse the complete 
focus interval Az. During the continuous stage movement through focus, successive 
images of the preparation are captured at 40 ms intervals or at any other standard video 

20 rate. The focus score of each captured image is then calculated. In accordance with a 
preferred embodiment of the present invention the image buffer is re-used for the next 
video frame, necessitating only two image memory buffers to be active at any time. One 
of the buffers is used to provide the data necessary for the focus score calculation of the 
previously captured image, while the other is used for capturing the next image. Hence, 

25 calculation of the focus score is preferably performed within one video frame time. 

As soon as the stage has reached the end of the focus interval, timing is stopped 
at / = id ms. An estimation of the focus curve is obtained from the focus score results for 
the complete focus interval. The global optimum in the estimate for the focus curve 
represents the focal plane to be used as the final focusing position of the specimen. Each 

30 z-position is calculated from the time at which the corresponding image was captured. 

When linear movement of the stage is assumed, the position at which the image at time // 
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is taken corresponds to 

where id represents the travel duration, Az is the focus interval, and is the start 
position (position at / = 0 ms). 
5 It has been found that it is safe to assume that the focus curve is parabolic around 

the focal plane. From this a high focus precision can be achieved by quadratic 
interpolation. When assuming linear stage movement, or z = vt + z^, the focus curve 
around the focal plane can be approximately by 

s(t) = c + bt + ap- (4) 

10 The exact focus position is obtained by fitting a parabola through the detected optimum 
and its neighboring measurements. Consider the detected optimum s(1 0 ) = s 0 at time r = 
t 0 . The time axis may be redefined such that the detected optimum is at time / = 0. Then, 
neighboring scores are given by (SnJj and (s^tp), respectively. Solving for a, b and c 
gives 

2 2 2 2 
15 c = j<j,A = ^ ^ — = ^ ^ — (5) 

The peak of the parabola, and thus the elapsed time to the focus position, is given by 



2 2 2 2 

b sot n - s p 1 n - s otp+ s ntp 

2a 2\s 0 l n -spt n -s 0 tp + s n 1p) 

20 The focal plane is at a position given by: 

if 

zf =^Az + z m i n (7) 

to which the specimen is moved, taking the backlash correction into account. 

The depth of field of an optical system is defined as the axial distance form the 
focal plane over which details stili can be observed with satisfactory sharpness. The 
25 thickness of the slice which can be considered in focus is then given by : 
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NA Z 

where n is the refractive index of the medium, X the wavelength of the used light, and NA 
the numerical aperture of the objective. The focus curve is sampled at Nyquist rate when 
measured at r d intervals. Common video hardware captures frames at a fixed rate. Thus 
5 the sampling density of the focus curve can only be influenced by adjusting the stage 
velocity to travel z d pm per video frame time. 

The present invention also includes additional method steps if the focus score is 
not good enough after filtering with one value of o or if the attempt to fit the focus scores 
to a polynomial, e.g. a parabola, is not successful, e.g. there is no pronounced maximum 

10 within the focus scores. In accordance with one embodiment of the present invention the 
filtering step may be repeated with another value of o. This second value of o may be set 
automatically by the host processor 9. For instance, if a suitable focus score or maximum 
thereof is not obtained, the host processor 9 may increase the value of o and repeat the 
focusing steps. Alternatively or additionally, if a suitable focus score or maximum of the 

1 5 focus score is not obtained, a field of the specimen adjacent the field used for the focus 
attempt may be chosen and the focusing steps in accordance with the present invention. 
This process may be extended by electronically setting different values of o at the new 
field position and repeating the focusing steps again. After a predetermined number of 
failures, the autofocusing procedure may be terminated and the relevant field or specimen 

20 may be flagged in some way to indicate that focusing was not possible. 

Preferably, the spatial extent o of the gradient filter in accordance with the present 
invention is settable in order to optimize noise rejection and object location. In 
accordance with preferred embodiments of the present invention the operator may input a 
spatial extent for the focusing process. For example, the operator may input a value of 3 

25 micron to the host processor 9 of Fig. 2. The processor calculates the value of o based on 
equation 2 and uses this for the gradient filter. Alternatively, the operator inputs a range 
into the host processor 9, e.g. 2 to 5 micron. The host processor 9 is programmed to 
select a value from this range, e.g. the mid- value of 3.5 or the lowest value of 2 microns 
and to determine the value of o from equation 2 using this value. If focusing is not 

30 successful, i.e. the focus score values are not adequate or a maximum is not found, the 
host processor 9 may sequentially and automatically select other extent values within the 
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range input by the operator. Alternatively, the processor may select values for a without 
operator input. For example, it may start with a default value and select other values if 
this proves unsuccessful. The exact value of o is not critical and the digital filter in 
accordance with the present invention is robust. As shown from Fig. 5 the same focusing 
5 position is identified over a range of a values. 

A further embodiment of the present invention includes calculating the focus score 
for each image at each Z position with a plurality of o values e.g. a } o 2 ... On. This may 
be done in parallel. The limit on the number of o values N may be limited only by the 
capability of the host processor 9 to calculate the focus scores within a reasonable time. 

10 After the focus scores 1-N have been calculated for all the images, each set of scores 
associated with one o value is fitted to a suitable curve, e.g. polynomial to obtain the 
focus position as described above. The result is a set of N focus positions or less if some 
of the a values did not result in a focus position. Various algorithms may be applied to 
obtain the best focus position from the obtained focus positions. For instance, it is 

1 5 anticipated that several of the focus positions will be the same or nearly the same. Hence, 
there may be a clustering of the focus positions. The cluster may be extracted by digital 
processing techniques, e.g. eliminating extreme focus positions more than 3 times the 
standard deviation (least squares) from the average focus position calculated from the set 
of focus positions. Then an average best focus position is selected as the focus position. 

20 The number of different o values N does not generally have to be large. As the 

value of o determines which size of element is selected by the filtering operation for 
autofocusing, small changes in a will not usually change the determined focus point very 
much (see Fig. 5). Hence, widely spaced o can be used. Secondly, it not desirable for the 
filter to select large or small elements of the image which are not related to the desired 

25 focused object. Hence, in accordance with an embodiment of the present invention it is 
preferred if the range of the values of a is settable y the operator. In this way, the 
operator has some control over what elements of the image will be selected for 
autofocusing. 

To calculate the focus score within the video frame time for current sensors and 
30 computer systems, a simplification of the focus function eq. 1 is preferably considered 
with present day computing speeds but the present invention is not limited thereto. For 
biological preparations, details may be distributed isotropically over the image. In 
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accordance with an embodiment of the present invention the response of the filter in one 
direction is used for the determination of the focal plane. Further computation time can be 
saved by estimating the filter response from a fraction of the scan lines in the image. 
Then, the focus function may be given by 

5 H°) = -~ £[/(*>>')* G X (x^a)} 2 (9) 

For example, each sixth row (L = 6) may be applied. A recursive implementation of the 
Gaussian derivative is used, for which the computation time is independent of the value of 
o. The calculation time could be kept under 40 ms for all computer systems used, even 
when the system was running other tasks simultaneously. Comparison between the focus 

10 curve in two dimensions for the whole image eq. 1 and the response of eg. 9 reveals only 
by marginal differences for all experiments. 

For the acquisition of multiple aligned images from large, flat preparations, the 
variation in focus position is assumed small but noticeable at high magnification. Proper 
acquisition of adjacent images can be obtained by focusing a few fields. Within the 

15 preparation, a procedure in accordance with an embodiment of the present invention 
starts by focusing the first field. Fields surrounding the focuses field are captured, until 
the next field to be captured is a given distance away from the initially focused field. 
Deviation from best focus is now corrected for by focusing over a small interval. The 
preparation is scanned, keeping track of focus position at fields further away than a given 

20 distance from the nearest of all the previously focused fields. The threshold distance for 
which focusing is skipped depends on the preparation flatness and magnification, and has 
to be empirically optimized for efficiency. Fields that have been skipped for focusing are 
positioned at the focus level of the nearest focused field. Small variations in focus 
position while scanning the preparation are corrected during acquisition. 

25 The autofocus algorithm in accordance with the present invention has been tested 

in the following applications: a) quantitative neuronal morphology, b) time-lapse 
experiments of cardiac myocyte dedifferentiation, c) immunohistochemical label detection 
in fixed tissue d) C. Elegans GFP-VM screening, and e) immunocytochemical label 
detection in fixed cells. Each of these applications is described below. The software 

30 package SCIL-Image version 1 A (16) (TNO-TPD, Delft, The Netherlands) is used for 
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image processing, extended with the autofocus algorithm and functions for automatic 
stage control and image capturing in accordance with the present invention. 

Quantitative Neuronal Morphology in Bright-field Mode 

5 Morphological changes of neurons were automatically quantified as described in 

"Sodium butyrate induces aberrant tau phosphorylation and programmed cell death in 
human neuroblastoma cells", Brain Res. 1995, 688, pages 86-94. Briefly, PC 12 cells were 
plated in poly-L-lysine (Sigma, St. Louis, MO) coated 12-well plates. In each well 5 x 10 4 
cells were seeded. After 24 hours the cells were fixed with 1% glutaraldehyde for 10 
1 0 minutes. Then the cells were washed twice with distilled water. The plates were dried in 
an incubator. 

The plates were examined with objective 5 x NA 0.15 Plan-Neofluar, in bright- 
field illumination mode on an Axiovert 10 microscope (Carl Zeiss, Oberkochen, 
Germany). A scanning stage (stage and MAC4000 controller, Marzhauser, Wetzlar, 

1 5 Germany) was used for automatic position control. At power on, the stage was calibrated 
and an initial focus level was indicated manually. The camera used was an MX5 
(Adaptec, Eindhoven, The Netherlands) 780 x 576 video frame transfer CCD with pixel 
size 8.2 x 1 6.07 |im 2 , operating at room temperature with auto gain turned off. Adjacent 
images were captured by an lndy R4600 132 MHz workstation (Silicon Graphics, 

20 Mountain View, CA). As a result, an 8 x 8 mosaic image covering an area of 6.7 x 6.7 
mm 2 was electronically stored on disk for each well. Prior to the acquisition of the well, 
autofocusing at the center of the scan area was performed. The smallest details to focus 
were the neurites, which were about 3 pixels thick, yielding a = 1 .0 (eq. 2). Variability in 
the z-position of the wells turned out to be within 500 jam, which is taken as the focus 

25 interval. The wavelength of the illumination was about 530 nm, resulting in 23.6 jam 

depth of field (eq. 8). Therefore, stage velocity was reduced to 24.7 \im per video frame 
(10,000 steps per second) during focusing. Due to the low magnification, backlash 
correction was not necessary. 

30 Quantitative Neuronal Morphology - results 

Fig. 4a shows the average focus curve for 180 wells, which were all accurately 
focused according to an experienced observer. The measured focus curve with the lowest 
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maximum score (peak at 0.004) is at a field containing only some dead cells. The 
variation in focus score is due to the different number of cells, and their morphology. The 
local maximum beneath focus is caused by a 1 80° phase shift in the point-spread function 
of the optical system. 

5 The time needed for focusing was 1.7 seconds per field, consuming 7.5% of the 

time to acquire one 12-weIl plate (4.5 minutes). For these thoroughly stained 
preparations, the autofocus method was able to focus all fields. Even for fields containing 
only a few dead cells, the focal plane was accurately determined. 



10 Cardiac Myocyte Dediflferentiation in Phase Contrast Mode 

Cardiac myocytes were isolated from adult rats (ca. 250 gram) heart by 
collagenase as described in Donck LV, Pauwels, PL Vandeplassche G. Borgers M, 
"Isolated rat cardiac myocytes as an experimental model to study calcium overload: the 
effect of calcium-entry blockers". Life Sci 1986;38:765-772. The cell suspension 

15 containing cardiomyocytes and fibroblasts was seeded on lazminin coated plastic petri 
dishes, supplied with Ml 99 and incubated for one hour. Thereafter, unattached and/or 
dead cells were washed away by rinsing once with Ml 99. The petri dishes were filled 
with Ml 999 + 20% fetal bovine serum and incubated at 37°C. 

The petri dishes were examined with objective 32 x NA 0.4 Achrostigmat Phase 

20 1, in phase contrast mode on an Axiovert 35 microscope (Carl Zeiss, Oberkochen, 

Germany). During the experiment, ambient temperature was maintained at 37°C. Time- 
lapse recordings (15 hours) were made in 6 manually selected fields, one in each of the 6 
petri dishes. A scanning stage (stage and MAC4000 controller, Marzhauser, Wetzlar, 
Germany) visited the selected fields at 120 second intervals. Fields were captured using a 

25 CCD camera (TM-765E, Pulnix, Alzenau, Germany). They were added to JPEG 
compressed digital rnoyies (Indy workstation with Cosmo compressor card, SGI, 
Mountain View, CA), one for each selected field. Autofocusing was applied once per 
cycle, successively refocusing all the fields in 6 cycles. The smallest details to focus were 
the cell borders, which were less than 4 pixels thick, yielding o = 1.0 (eq. 2). Variability 

30 in the z-position between focus events was expected to be within 100 pirn, which was 
taken as the focus interval. The wavelength of the illumination was about 530 nm, 
resulting in 3.3 [im depth of field (eq. 8). Therefore, stage velocity was reduced to 2.5 
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l_im per video frame (1,000 steps per second) during focusing. 

Cardiac Myocyte DedifTcrentiation - results 

Fig. 4b shows the average focus curve (75 events) for one field of the time lapse. 
5 AJ1 the fields were perfectly focused during the 15 hours of recording, according to 
experienced observers. The variation in focus score was due to the change in image 
content, caused by movement of fibroblasts and the dedifierentiation of the myocytes. 
The axial drift from the initial focal plane during the recording of the six positions varied 
from 3 jam up to 27 pm. The time needed for focusing was 2.8 seconds per field. 
10 Rat cardiomyocytes are known to dedifferentiate spontaneously in culture, i.e., 

they spread out, flatten and develop pseudopodia-like processes. Despite these changes in 
image content during the experiment, none of the time lapse movies was out of focus any 
time. 

15 Jmmunohistochemical Label Detection in Bright-field Mode 

Sections of the amygdala of mice injected with a toxic compound were cut at 15 
pm thickness through the injection site. They were subsequently immunostained for the 
presence of the antigen, using a polyclonal antibody (44-136, Quality Control 
Biochemicals Inc., Hopkinton, MA) and visualized using the chromogen DAB. 

20 Four microscope slides (40 brain slices) were mounted on the stage of an 

Axioskop microscope (Carl Zeiss, Oberkochen, Germany) and examined with objective 
2.5 x NA 0.075 Plan-Neofluar, in bright-field illumination mode. A scanning stage (stage 
and MC2000 controller, Marzhauser, Wetzlar, Germany) was used for automatic position 
control. Adjacent images were captured (Meteor/RGB Framegrabber, Matrox, Donval, 

25 Quebec, Canada in an Optiplex GXi PC with Pentium 200 MHz MMX, Dell, Round 

Rock, TX) by use of an MX5 CCD camera (Adaptec, Eindhoven, The Netherlands). As a 
result, mosaics of complete brain slices were stored electronically on disk. Prior to 
acquisition, autofocusing at approximately the center of the brain slice was performed. 
The smallest detail to focus was the tissue structure, which was about 3 pixels thick, 

30 yielding a = 1 .0 (eq. 2). Variability in the z-position between the glass slides turned out to 
be within 1000 pm, which was taken as the focus interval. The wavelength of the emitted 
light was about 530 nm, resulting in 94 fim depth of field (eq. 8). Therefore, stage 
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velocity was reduced to 98.7 |am per video frame (40,000 steps per second) during 
focusing. Due to the low magnification, backlash correction was not necessary. 



Immunohislochemical Label Detection - results 

5 Fig. 4c shows the average focus curve for the rat brain slices. From the 100 fields 

examined, 2 fields contained not enough contrast for focusing. These are not included in 
Fig. 4c. The variation in focus score was caused by the differences in contrast between 
the slices. 

The time needed for focusing was 1 .5 seconds per field, consuming 7% of the 
10 time to acquire one glass slide (3 minutes). The autofocus method was able to focus 98% 
of the fields. The contrast in the remaining fields was to low for accurate focusing. 

C Elegans GFP-VM Screening in Fluorescence Mode 

Individual C. Elegans worms transgenic for GFP expressing vulval muscles (GFP- 

1 5 VM) were selected from stock, and one young adult hermafrodite (P 0 ) was placed in each 
of the 60 center wells of a 96- well plate (Costar, Acton, MA) filled with natural growth 
medium, and incubated for five days at 25°C to allow Fi progeny to reach adult stage. 

Before image acquisition, fluorescent beads (F-8839, Molecular Probes, Eugene, 
OR) were added to the wells as background markers for the focus algorithm. The well 

20 plate was examined with an objectives 40 x NA 0.6 Archoplan, in fluorescence mode on 
an Axiovert 135 microscope (Carl Zeiss, Oberkochen, Germany). A FITC filter (B, Carl 
Zeiss, Oberkochen, Germany) in combination with a 100W Xenbophot lamp was used to 
excite the green fluorescent protein (GFP). Images were captured (02 R5000 180 MHz 
workstation, Silicon Graphics, Mountain View, CA) using an intensified CCD camera 

25 (1C-200, PTI, Monmouth Junction, NJ). The scanning stage (stage and MC2000 

controller, Marzhauser, Wetzlar, Germany) was calibrated to capture adjacent images. 
Each of the selected wells was scanned and the adjacent images, completely covering the 
well, were stored electronically on disk. Variability in the z-position between the center of 
the wells turned out to be within 250 }im, which was taken as the focus interval. After 

30 autofocusing on the well center, deviation from best focus while scanning the well was 
corrected. It turned out that autofocusing over one-fifth the focus interval (50 jam) for 
fields further than 3 fields away from a focused field was sufficient. The fluorescent 
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spheres were 30 pixels in diameter, yielding o = 8.5 (eq. 2). The wavelength of the 
emitted light was about 530 nm, resulting in 1.47 p.m depth of field (eq. 8). The diameter 
of the fluorescent spheres was 1 5 |_im, which is much larger than z<j. Since the spheres 
were homogeneously stained, the smallest detail to consider in the z-direction was a 
5 cylindrically shaped slice through the spheres, where the cylinder height was determined 
by the horizontal resolution. Therefore, the stage velocity was reduced to approximately 
one third of the sphere diameter during focusing, yielding 4.94 \xm per video frame (2,000 
steps per second). Backlash correction offset was determined to be 15 jam. 



10 C. Elegans GFP-VM Screening -results 

Fig. 4d shows the average focus curve for 1786 fields of the C. Elegans screening. 
Fourteen of the 1800 fields did not contain fluorescent spheres, consequently accurate 
focusing was not possible. These are not included in Fig. 4d. The remaining fields were 
accurately focused according to an experience observer. The variation in focus score was 

1 5 caused by the different number of spheres and the presence or absence of worms in the 
focused fields. The time needed for focusing was 2.8 seconds for the first field in a well, 
and 1 . 1 seconds per field for keeping track of the focal plane within the well. 
Autofocusing consumed 12% of the time to acquire one 96-well plate (4.5 hours for 
28,000 images), which is reasonable given the time needed for the preparation. 

20 The images were highly degraded by the presence of random noise (SNR « 10 

dB) due to fluorescent bacteria and structural noise caused by earth loops in combination 
with the extremely sensitive CCD camera. The focus algorithm was able to find the 
correct focus position for all but 14 out of 1800 fields examined. Failure was caused by a 
shortage of relevant image information content. 

25 

Immunocytochemical Label Detection in Fluorescence Mode 

Human fibroblasts were seeded in a 96-well plate (Costar, Acton, MA) at 7000 
cells per well, in 2% FBS/Optimem. The cells were fixed and permeabilized by adding 2% 
paraformaldehyde and 0.5% Triton-X-100 (Sigma, St. Louis, MO). The reaction was 
30 stopped after 30 minutes by washing with PBS, addition of 5% NGS during 60 minutes, 
and washing with 0.2% BSA/PBS for 5 minutes. A dilution of 1/100 primary antibody 
rabbit anti human NF- K B (p65) (Santa Cruz Biotechnology, Santa Cruz, CA) in 0.2% 
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BSA/PBS was added to each well for 2 hours at 37°C After washing 3 times with PBS 
for 10 minutes, Cy3 labeled sheep ami rabbit (Jackson, Uvert-Grove, PA) dilution (1/2 
glycerol, 1/250 0.2% BSA/PBS) was added to each well for 60 minutes at 37°C. After 
washing with 0.2% BSA/PBS for 1 5 minutes and repeated washing with PBS, and 
5 nuclear counter staining with Hoechst 33342 (Molecular Probes, Eugene, OR), the cells 
were ready for quantitative evaluation. 

Well plates were examined with an objective 40 x NA 0.6 Achroplan, in 
fluorescence mode on an Axiovert 135 microscope (Carl Zeiss, Oberkochen, Germany). 
A DAPI/F1TC/TRJTC filter (XF66, Omega Optical, Brattleboro, VT) in combination 

10 with a 100W Xenophot lamp was used to excite the cells. A scanning stage (stage and 
MC2000 controller, Marzhauser, Wetzlar, Germany) was used for automatic position 
control. Adjacent images were captured (02 workstation R5000 180MHZ, Silicon 
Graphics, Mountain View, CA) using an intensified CCD camera (1C-200, PT1, 
Monmouth Junction, NJ). As a result, two 5x5 mosaic images, one for the nuclei and 

15 one for the immuno signal, was stored on disk for each well. The covered area per well 
was 1 .2 x 1.2 mm 2 . Prior to the acquisition of each mosaic image, autofocusing at 
approximately the center of the scan area was performed. The smallest details to focus 
were the nuclei, which were minimally 30 pixels in diameter, yielding o = 8.5 (eq. 2). 
Variability in the z-position of the wells turned out to be within 250 ^im, which was taken 

20 as the focus interval. The wave length of the emitted light was about 450 nm (nuclei) and 
600 nm (immuno signal), resulting in 1.25 \xn\ and 1.67 jam depth of field (eq. 8), 
respectively. Cell thickness was about 5-15 jam, much larger than z d . Therefore, the stage 
velocity was reduced to 4.94 jam per video frame (2,000 steps per second) during 
focusing. Backlash correction offset was determined to be 15 \im. 

25 

Immunocytochemical Label detection - results 

Fig. 4e shows the average focus curve for the nuclei in the immunocytochemical 
label detection. All the 150 fields were accurately focused according to an experienced 
observer. The variation in focus score was due to the different number of cells present in 
30 each field. The average focus curve as measured for the immuno signal is shown in Fig. 
4f. From the 1 50 fields examined, 2 fields were completely saturated due to preparation 
artifacts, causing the focus algorithm to fail. These fields are not included in Fig. 4f. the 
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remaining fields were accurately focused. The time needed for focusing was 2.8 seconds 
per field, consuming 14% of the time to acquire one 96-well plate (20 minutes). 

The signal to noise ratio was estimated to be about 10 dB for the nuclei, and 4dB 
for the immuno signal. Despite the noise, the autofocus method was able to focus all but 
5 two out of 300 fields. Failure was caused by a shortage of relevant image information 
content. 



Comparison of Performance with Small Derivative Filters 

In order to evaluate the effect of the scale o in the estimate for the focus score, 

1 0 experiments with a fixed (non- select able) a = 0.5 were performed. 

For the quantitative neuronal morphology, accurate focusing with a — 0.5 was not 
possible for 1 out of 24 fields ]n this case, the algorithm focused on the reversed contrast 
image. Application of the small scale in focusing of the cardiac myocyte dedifferentiation 
failed whenever fungal contamination at the medium surface occurred, which was taken 

1 5 as the local plane. Taking o = 1.0 solved this problem, that is by focusing persistently on 
the myocytes. Focusing with o = 0.5 on the immunohistochemical label detection resulted 
in focusing on dust particles at the glass surface for 5 out of 24 fields. 

For the fluorescence applications, accurate focusing was not possible with o = 
0.5, due to the small signal to noise ratio. Experiments taken with o = 0.75 resulted in 

20 inaccurate focusing for 18 out of 30 fields for the C. Elegans GFP-VM screening. 

Further, the algorithm was not able to focus accurately on 13 out of 30 fields for the 
nuclei in the immunocytochemical label detection, and failed for 17 out of 30 fields on the 
immuno signal. 

The effect of the scale o results in robustness against noise and artifacts. A larger 
25 scale results in robustness against phase reversion (quantitative neuronal morphology), 
fungal contamination at the medium surface (cardia myocyte dedifferentiation), dust on 
the glass surface (immunohistochemical label detection) and noise (the fluorescence 
applications). From these results it can be inferred that the performance of small 
differential filters, as used conventionally, is poor given the number of inaccurately 
30 focused images for o = 0.5 or o = 0.75. Further, it can be inferred that the scale o should 
preferably be at least 1 . This is equivalent to 3 pixels. Hence, a preferred range for the 
scale o in accordance with the present invention is from at least 3 pixels up to the size of 
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the image itself 

For the different applications, the chosen focus interval was effectively used for 
about 30%. The focus interval is preferably not taken too narrowly to ensure that the 
focal plane is inside the interval, regardless of the manual placement of the preparations. 
5 The time needed for the autofocus algorithm varied from 1.5 up to 2.8 seconds 

for current sensors and computer systems, which is in the same time range as trained 
observers. Focus time is determined by the depth of field and the video frame time, both 
of which can be considered as given quantities, and by the size of the focus interval. 
Further reduction of focus time can be achieved by a smaller focus interval, on the 

10 condition that the variability in preparation position is limited. When positional variability 
is low or well known, the focus interval Az can be reduced to exactly fit the variability. 
For the applications given, the focus time can be reduced up to a factor 3 in this way. 

Failure of the autofocus algorithm due to a shortage of image content can be well 
predicted. If the focal plane is inside the focus interval, there should be global maximum 

1 5 in the estimate of the focus curve. Comparing the maximum focus score s 0 with the 

highest of the focus scores at the ends of the focus interval, s* = max(s(0), s(t d )) which 
are certainly not in focus, determines the signal content with respect to noise. When the 
maximum score does not exceed significantly the focus scores at the ends of the interval, 
or (so - s c )/s e < a, the found focus position should be rejected. In this case, focusing can 

20 better be based on a neighboring field. For the reported results, a threshold of a = 10% 
safely predicts all failures. Accordingly, in accordance with the present invention a 
specimen may be focused correctly, or if the specimen cannot be focused correctly this 
can be determined by the system from the focus scores and this sample and/or its image 
can be labeled/flagged that focusing was inadequate automatically. This provides a 

25 significant advantage as the operator may then identify these specimens and carry out 

other forms of focusing as required, e.g. manually without the time-consuming necessity 
to check every image to see if any are out of focus. 
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CLAIMS 

1. A method of autofocus of an optical instrument for viewing an object and having an 
auto-focusing mechanism, comprising the steps of: 

5 step 1 : acquiring a first digital image of the object through the optical instrument, 

the first digital image comprising a plurality of pixels having pixel values; 

step 2: applying a digital gradient filter to at least some of the pixel values of the 
first digital image to obtain a focus score for the first digital image; wherein the digital 
gradient filtering step includes a smoothing operation having a settable spatial extent. 

10 

2. The method of claim 1, wherein the spatial extent of the smoothing function is manually 
and/or electronically settable. 

3. A method of autofocus for an optical instrument for viewing an object and having an 
15 auto-focusing mechanism, comprising the steps of: 

step 1 : acquiring a first digital image of the object through the optical instrument, 
the first digital image comprising a plurality of pixels having pixel values; 

step 2: applying a digital filter to at least some of the pixel values of the first digital 
image to obtain a focus score for the first digital image; wherein the digital filter is defined 
20 by a mathematical smoothing function having a negative and positive lobe around the 
origin thereof, the mathematical smoothing function having only one zero crossing and 
being limited in spatial extent in that it extends over a distance smaller than or equal to the 
image size and extends ?.t least over three pixels either side of a pixel whose value is being 
filtered. 

25 

4. The method according to any previous claim, further comprising: 

step 3: moving the object relative to the optical instrument along the optical axis 
thereof and acquiring a second digital image and a second focus score therefor in 
accordance with the method of steps 1 and 2; 
30 step 4: continue moving the object relative to the optical instrument along the 

optical axis thereof in the same direction in accordance with steps 1 to 3 to acquire at least 
three digital images and first to third focus scores associated therewith; and 

step 5: determining from the first to third focus scores a focus position for the 
object and moving the object and/or the optical instrument to this position. 
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5. The method according to claim 1 or 2, further comprising: 

step 3: determining a first plurality of focus scores for the first digital image using 
the digital gradient filler with a first plurality of spatial extents by applying for each spatial 
extent the method steps 1 and 2 ; 

step 4: moving the object relative to the optical instrument along the optical axis 
thereof and acquiring a second digital image and a second plurality of focus scores therefor 
in accordance with the method of step 3; 

step 5: continue moving the object relative to the optical instrument along the 
optical axis thereof in the same direction in accordance with steps 1 to 3 to acquire at least 
three digital images and first to third pluralities of focus scores associated therewith; and 

step 6: determining from the first to third plurality of focus scores a focus position 
for the object and moving the object and/or the optical instrument to this position. 

6. The method according to any previous claim, wherein the optical instrument is a 
microscope. 

7. The method according to claim 4 , 5 or 6, wherein the determining step includes fitting 
the focus scores to a polynomial function and moving the object and/or the optical 
instrument to a position related to a maximum of the polynomial function. 

8. The method according to any of the previous claims, wherein the digital filtering 
function is a one or two-dimensional function. 

9. The method according to any previous claim, wherein the digital filtering function is a 
Gaussian function. 

10. The method according to any of the previous claims, further comprising the step of 
selecting the spatial extent of the digital filtering function. 

1 1. An optical instrument for viewing an object and having an auto-focusing mechanism, 
the optical instrument being adapted to acquire a first digital image of the object through 
the optical instrument, the first digital image comprising a plurality of pixels haying pixel 
values; and the auto- focusing mechanism having a digital gradient filter to filter at least 
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some of the pixel values of the first digital image and to obtain a focus score for the first 
digital image, wherein the digital gradient filter includes a smoothing function having a 
settable spatial extent. 

5 12. The opiical instrument of claim 1 1, wherein the spatial extent of the smoothing 
function is manually and/or electronically settable. 

13. An optical instrument for viewing an object and having an auto-focusing mechanism, 
the optical instrument being adapted to acquire a first digital image of the object through 

10 the optical instrument, the first digital image comprising a plurality of pixels having pixel 
values; and the auto-focusing mechanism having a digital filter to filter at least some of the 
pixel values of the first digital image to obtain a focus score for the first digital image 
wherein the digital filter is defined by a mathematical smoothing function having a 
negative and positive lobe around the origin thereof the mathematical smoothing function 

15 having only one zero crossing and being limited in spatial extent in that it extends over a 
distance smaller than or equal to the image size and extends at least over three pixels either 
side of a pixel whose value is being filtered. 

14. The optica] instrument according to any of claims 1 1 to 13, further comprising: a drive 
20 device for moving the object relative to the optical instrument along the optical axis 

thereof 

15. The optical instrument according to any of claims 1 1 to 14, the instrument being 
further adapted for determining from a plurality of focus scores for a plurality of images a 

25 focus position for the object. 

16. The optical instrument according to claim 15 further adapted for fitting the plurality of 
focus scores to a polynomial function and determining the focus position as a position 
related to a maximum of the polynomial function. 

30 

17. The optical instrument according to 15, the instrument being adapted to determine for 
each image a plurality of focus scores using a plurality of spatial extents for the digital 
filter. 
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1 8. The optical instrument according to any of the claims 1 1 to 17, wherein the digital 
filtering function is a one or two-dimensional function. 

19. The optica] instrument according to any of claims 1 1 to 18, wherein the digital filtering 
5 function is a Gaussian function. 

20. The optical instrument according to any of claims 1 1 to 19, wherein the optical 
instrument is a microscope. 

10 21. The optical instrument according to any of the claims 13 to 20, wherein the extent of 
the digital filtering function is manually and/or electronically settable. 

22. An auto-focusing mechanism for an optical instrument, the optical instrument being 
provided for viewing an object and for acquiring a digital image of the object, the digital 
15 image comprising a plurality of pixels having pixel values: the mechanism comprising: a 
digital gradient filter to filter at least some of the pixel values of the digital image to obtain 
a focus score for the digital image, wherein the digital gradient filter includes a smoothing 
function having a settable spatial extent. 

20 23. The autofocus mechanism according to claim 22, wherein the spatial extent of the 
smoothing function is electronically and/or manually settable. 

24. An auto-focusing mechanism for an optical instrument, the optical instrument being 
provided for viewing an object and for acquiring a digital image of the object, the digital 

25 image comprising a plurality of pixels having pixel values; the mechanism comprising: 
a digital filter to filter at least some of the pixel values of the digital image to obtain a 
focus score for the digital image wherein the digital filter is defined by a mathematical 
smoothing function having a negative and positive lobe around the origin thereof, the 
mathematical smoothing function having only one zero crossing and being limited in 

30 spatial extent in that it extends over a distance smaller than or equal to the image size and 
extends at least over three pixels either side of a pixel whose value is being filtered. 

25. The mechanism according to any of claims 22 to 24, wherein the digital filtering 
function is a one or two-dimensional function. 
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26. The mechanism according to any of claims 22 to 25, wherein the digital filtering 
function is a Gaussian function. 

27. The mechanism according to any of claims 22 to 26 r comprising the mechanism being 
adapted for determining from a plurality of focus scores for a plurality of images a focus 
position for the object. 

28. The mechanism according to claim 27, further adapted for fitting the plurality of focus 
scores to a polynomial function and determining the focus position as a position related to 
a maximum of the polynomial function. 



0O75709A1 I > 




BNSDOCID: <WO 0075709A1 J_> 




BNSDOCID: <WO 0075709A1 J_> 



WO 00/75709 



PO7EP00/04987 



4/13 




BNSDOCID: <WO 007S709A1_I_> 



WO 00/75709 



PCT/EP00/04987 




BNSDOCID: <WO 0075709A1_I_> 




BNSDOC1D: <WO O075709Al_L> 



WO 00/75709 



PCT/EPOO/04987 



7/13 



o 




*^-C0C\Ji— Or-C\JC0^f 

oooo oooo 

• • • • • • _ ■ _ ■ 

oooo oooo 

I I I I 



0075709AU > 



WO 00/75709 



8/13 



PCT/EPOO/04987 




BNSDOCID: <WO 0075709A1_I_> 



WO 00/75709 



PCT/EP00/04987 



9/13 



0.8 



0.6 



0.4 - 



0.2 - 



mean 



mm — 
max 




-250 -200 -150 -100 -50 0 50 

z-position [urn] 



100 150 200 250 



0.8 - 



0.6 - 



0.4 - 



0.2 - 




-40 



-20 0 

z-position [umj 



BNSDOCID: <WO 0075709A1_L> 



WO 00/75709 PCT/EP00/04987 

10/15 




0 1 1 1 1 1 ' 

"400 -200 0 200 400 

z-position [urn] 



fr 5 l+c 



BNSDOCID: <WO 0075709A1_L> 



I. 



PCT/EP00/O4987 




-60 



0.8 - 



0.6 - 



0.4 - 



0.2 - 



-60 



-40 -20 0 20 

z-position [urn] 



-40 



f?S- 43 




-20 0 20 

z-position [um] 



65. 4 ^ 



BNSDOCID: <WO 0076709A1 J_> 



WO 00/75709 PCT/EP00/04987 

12/13 

1 t— i 1 i '• 1 

mean 

min 

max 

0.8 - 



0.6 - 



0.4 - 




-60 -40 -20 0 20 40 60 



z-position [um] 

fr 5 . ^ 



BNSDOCID: <WO 0075709A1_I_> 




BNSDOCtD: <WO 0075709A1J_> 



INTERNATIONAL SEARCH REPORT 




Inte. onal Application No 

PCT/EP 00/04987 | 


a r»i Accicir ATtnu AC *^URJEf*T MATTER 

IPC 7 G02B21/24 




Accordina to International Patent Classification (IPC) or to both national classification and IPC 




B. FIELDS SEARCHED f 


Minimum documentation searched (classification system followed by classification symbols) | 



IPC 7 G02B 



Documentation searched other than minimum documentation to the extent tnat such documents are included in the fields searched 



Electronic data base consulted dunng the international search (name of data base and. where practical, search terms used) 

EPO-Internal, WPI Data, PAJ, INSPEC 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 0 Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



US 5 790 710 A (G0UGH DAVID A ET AL) 

4 August 1998 (1998-08-04) 

column 9, line 12 -column 10, line 26 

DE 197 26 696 A (JENOPTIK JENA GMBH) 
7 January 1999 (1999-01-07) 
column 6, line 22 - line 53 



US 4 203 031 A (G0CH0 NAGAHIR0 
13 May 1980 (1980-05-13) 
column 2, line 16 - line 36 



ET AL) 



1,3,11, 
13,22,24 



1,3,11, 
13,22,24 



1,3,11, 
13,22,24 



□ 



Further documents are listed in the continuation of box C. 



Patent family members are listed in annex. 



* Special categories of cited documents : 

"A" document defining the general state of the art which is not 

considered to be of particular relevance 
*E" earlier document but published on or after the international 

filing date 

"L" document which may throw doubts on priority claimCs) or 
which is cited to establish the publication date of another 
citation or other special reason (as specified) 

"O" document referring to an oral disclosure, use, exhibition or 
other means 

m P m document published prior to the international filing date but 
later than the priority date claimed 



T" later document published after the international filing date 
or priority date and not in conflict with the application but 
cited to understand the principle or theory underlying the 
invention 

"X" document of particular relevance: the claimed invention 
cannot be considered novel or cannot be considered to 
involve an inventive step when the document is taken alone 
*Y" document of particular relevance; the claimed invention 

cannot be considered to involve an inventive step when the 
document is combined with one or more other such docu- 
ments, such combination being obvious to a person skilled 
in the art- 
's • document member of the same patent family 



Date of the actual completion of the international search 

26 July 2000 


Date of mailing of the international search report 

01/08/2000 


Name and mailing address of the ISA 

European Patent Office. P.B. 5818 PatentJaan 2 
NL - 2280 HV Rijswifr 
Tel. (+31-70) 340-2040, Tx. 31 651 epo nl. 
j Fax: (+31-70) 340-3016 


Authorized officer 

Sarneel , A 



Form PCT/ISA/210 (socood sheet) (July 1992) 



BNSDOCID: <WO_ 



_O0757O9A1_l_> 



INTERNATIONAL SEARCH REPORT 



■nlormalion on patent lamily members 



Intei snal Application No 

PCT/EP 00/04987 



Patent document 
cited in search report 



Publication 
date 



Patent family 
member(s) 



Publication 
date 



US 5790710 



04-08-1998 



US 

AT 

AT 

AU 

CA 

DE 

DE 

DE 

DE 

EP 

EP 

ES 

ES 

JP 

WO 

US 

US 



5932872 
177539 
184113 
2960395 
2192986 
69508248 
69508248 
69511903 
69511903 
0769159 
0834758 
2131838 
2140178 
10502466 
9601438 
5548661 
5856665 



A 

T 

T 

A 

A 

D 

T 

D 

T 

A 

A 

T 

T 

T 

A 

A 

A 



03- 08- 
15-03- 
15-09- 
25-01- 
18-01- 

15- 04- 

04- 11- 

07- 10- 
20-04- 
23-04- 

08- 04- 
01-08- 

16- 02- 
03-03- 
18-01 
20-08 

05- 01 



1999 
1999 
1999 
1996 
1996 
1999 
1999 
1999 
2000 
1997 
•1998 
1999 
■2000 
-1998 
-1996 
-1996 
-1999 



DE 


19726696 


A 


07-01-1999 


WO 
EP 


9859235 
0991934 


A 
A 


30-12-1998 
12-04-2000 


US 


4203031 


A 


13-05-1980 


JP 
DE 


53116852 
2812593 


A 
A 


12-10-1978 
28-09-1978 



Form PCT/ISA/210 (patent tamfly annex) (July 1992) 
BNSDOCID: <WO 007S709A1 J_> 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



